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Abstract 

The  equality  and  inclusion  tests  for  two  sets  are  considered, 
and  it  is  shown  how  the  use  of  hashing  techniaues  can  speed 
up  these  operations.   Various  hash  functions  are  considered, 
and  their  relevant  properties  discussed  in  connection  with 
these  set  operations. 


1 1 1 


Introduction 

This  paper  is  concerned  with  a  particular  aspect  of  a 
very  general  problem,  namely:   what  is  the  best  way  to  repre- 
sent a  data-ob,ject  in  memory  in  order  to  minimize  the  time 
reouired  to  do  a  specified  operation.   It  is  clear  that, 
for  instance,  if  the  data-object  is  an  integer  and  we  are 
interested  in  testing  it  to  determine  if  it  is  odd  or  even, 
it  is  not  appropriate  to  represent  the  integer  as  a  coded  string 
of  letters  which  spell  out  the  number,  or  as  the  program  for 
a  Turing  machine  which  will  write  the  corresponding  number  of 
ones  on  its  tape.   Rather,  we  will  represent  it  in  binary,  so 
that  an  examination  of  the  least  significant  bit  will  suffice 
to  determine  its  parity.   Ideally,  every  data-object  should  be 
represented  in  the  optimum  way  for  each  operation  which  is  to 
be  performed  on  it.   This  would  give  a  highly  redundant  repre- 
sentation, so  for  small  data- objects  such  as  integers  it  is 
usual  to  choose  a  single  compromise  representation.   In  the 
case  of  large  data-objects  it  makes  more  sense  to  store  redun- 
dant information  if  it  is  not  too  space-consuming,  because  the 
time  to  recompute  this  information  may  be  considerable. 

We  can  usefully  distinguish  between  the  tyues  of  operation 
which  require  all  the  information  of  the  data-object,  such  as 
multiplication,  and  those  operations  which  require  only  part 
of  this  information.   The  parity  test  mentioned  above  is  an 
example  of  the  latter  type  of  operation,  since  the  required 
information  about  the  integer  is  only  a  single  bit. 


In  the  case  of  operations  which  dn  not  use  all  infor- 
mation such  as  those  mentioned  above,  it  is  clear  that  it  is 
preferable  in  some  cases  to  represent  the  data-objects  by  only- 
part  of  the  information  actually  available.   If  the  precise 
operations  are  not  known,  we  have  no  way  of  choosing  the 
optimum  representation,  but  it  is  still  a  reasonable  auestion 
to  ask  if  there  is  a  more  concise  representation  which  will 
be  adequate  for  certain  types  of  operations.   This  problem  is 
one  of  information  compression. 


UNORDERED  SETS 

Here  we  will  restrict  consideration  to  data-objects 
which  are  unordered  sets,  and  to  operations  which  test  one 
set  for  being  a  subset  of  the  other.   This  inclusion  test  is 
an  extremely  useful  test  which  occurs  in  a  number  of  disguises 
throughout  programming.   In  a  later  section  we  will  also  extend 
the  techniques  to  ordered  sets,  that  is,  sets  in  which  the 
order  of  the  elements  is  significant. 

We  first  consider  the  implementation  of  the  inclusion 
test  in  such  a  way  that  the  result  of  the  test  always  gives 
the  correct  ansi^er,  and  then  we  consider  how  the  test  may  be 
weakened  so  that  it  constitutes  a  necessary  but  not  sufficient 
condition  for  inclusion.   This  is  a  very  useful  form  of  test 
if  it  can  be  computed  faster  than  the  more  accurate  test,  because 
in  many  situations  the  inclusion  test  will  give  a  false  result 
most  of  the  time.   Note  that  the  form  of  test  we  are  advocating 
doe3  not  give  a  probabalistic  answer  purely  in  the  form: 

'the  probability  that  set  S,  is  included  in  set  S?  is  .03' 
but  gives  an  answer  either  of  the  above  form  or  of  the  form: 

'the  set  S-,  is  definitely  not  included  in  the  set  S?  * 
which  is  much  more  useful,  particularly  in  those  situations  in 
which  the  second  answer  is  given  in  most  cases. 


The  Inclusion  Test 

The  straightforward  way  of  implementing  the  inclusion 
test  is  to  select  an  element  of  one  set  S,  and  search  for  it 
in  the  other  set  S?,  repeating  this  operation  for  all  elements 
in  S, .   If  all  elements  are  found,  S-,  is  included  in  S„,  while 
if  any  element  is  not  found  S,  is  not  included  in  S2.   This  can 
easily  be  extended  to  test  for  equality  by  marking  those  elements 
in  S?  which  are  found  to  correspond  to  elements  in  S,,  the  sets 
being  equal  if  and  only  if  all  elements  of  S  get  marked.   If 
the  number  of  elements  in  the  two  sets  are  n,  and  n?,  the  time 
to  do  the  inclusion  test  is  proportional  to  n,  and  to  the  time 
required  to  locate  a  particular  element  in  Sp.   If  the  elements 
of  S?  are  stored  linearly  in  memory  in  a  random  order,  the 
inclusion  test  will  take  a  time  which  is  proportional  to  n.n   , 
while  if  the  elements  of  Sp  are  sorted,  a  binary  search  can  be 
used  to  reduce  this  to  n,log2n2.   If  both  sets  are  sorted,  then 
a  much  more  efficient  procedure  is  possible  which  is  proportional 
to  n?.   This  takes  the  form  of  a  search  for  the  first  element 
of  S,,  starting  at  the  beginning  of  Sg.   When  found,  the  search 
for  the  next  element  of  S,  starts  at  the  next  element  of  S2# 
Of  course,  the  time  required  to  sort  the  elements  has  to  be 
taken  into  account  also,  and  this  will  be  significant  if  a 
small  number  of  tests  are  made  with  each  set.   If  many  tests 
are  made  on  each  set,  it  is  worth  keeping  it  in  a  sorted  form. 


The  method  we  will  consider  here  can  be  used  to  speed 
up  the  eauality  or  inclusion  test  by  preceding  it  with  a 
weaker  but  computationally  faster  test  which  may  immediately 
exclude  the  possibility  of  equality  or  inclusion.   If  the  sets 
pass  the  preliminary  test,  then  the  more  rigorous  test  can  be 
applied.   If  the  probability  of  the  test  being  false  is  large, 
as  is  often  the  case  in  practice,  in  the  majority  of  cases 
the  preliminary  test  will  be  the  only  one  which  is  applied, 
resulting  in  a  considerable  gain  in  speed  even  when  the  sets 
are  sorted. 


Invertible  Set  Functions 

Consider  a  set  S  with  elements  e, ,  e2  ...  ,  e  ,  an<i  a 
function  H  on  such  a  set.   We  villi  consider  those  functions  H 
which  have  the  property  that  the  relationship  between  H(S-j) 
and  H(S?)  can  be  used  to  deduce  a  relationship  between  S.  and 
S?.   That  is,  the  function  H  will  be  renuired  to  preserve 
those  properties  of  the  set  which  are  relevant  to  the  relation- 
ship being  investigated.   We  can  regard  the  function  H  as  an 
information  compression  function. 

In  the  present  case  we  are  concerned  with  sets  of  items 
in  which  the  order  of  the  elements  is  irrelevant,  so  it  is 
reasonable  to  suppose  that  the  function  H  should  be  independent 
of  the  order  in  which  the  elements  are  specified.   If  we  write 
H(S)  as 

H( e-, ,  &2    •  •  •  >  en) 
this  implies  that 

H  ( .  . .  >6.  ...  ,  e  . ,  ...j  =  ti \ •  • .  >  e  . ,  ...  ,  e . ,  ...j 
Out  of  all  such  symmetric  functions  we  will  restrict  consideration 
to  those  which  satisfy 

H(elfe2,    ...  ,en)  =  h(e1)  9  E(e?,    ...  , en) 
for  same  h.   That  is 

H(e1,e2,  ...  ,en)  =  h(e1)  ©  h(e2)  ©  ...  ©  h(en^ 
where  the  operator  €  is  associative  and  commutative. 

In  what  follows  we  will  use  the  function  h  to  map  the 
elements  of  the  set  into  the  integers  ^-^^2    '"    jin'   In 


some  applications  we  may  have  duplicated  elements,  in  others 
we  may  assume  that  the  elements  are  unioue,  and  in  others  we 
may  assume  that  duplicated  elements  may  occur  but  that  the 
duplication  of  an  element  will  not  be  considered  to  change 
the  set. 

Let  us  first  consider  the  set  equality  test.   It  is  clear 
that  if  S,  and  S2  are  identical,  then  H(S,)  and  H(S.p)  are 
equal.   What  we  wish  to  do,  however,  is  to  be  able  to  make 
the  inference  in  the  reverse  direction.   We  can  certainly  do 
this  if  H  has  an  inverse;  that  is,  if  h  has  an  inverse,  and 
the  function: 

G(i1,i2,  ...  ,in)  -  i1  «  i2  ®   ...  $  in 
also  has  an  inverse. 

One  function  which  can  be  used  for  G  is: 

G(i1,i2,  ...  ,in)  =  pC^)  *  p(i2)  •  ...  *  p(in)      (A) 
where  p(i)  is  the  i'th  prime.   This  uses  the  fact  that  the 
prime  factors  of  an  integer  are  unique.   The  practical  disad- 
vantages of  this  scheme  are  that  for  large  sets  the  product 
becomes  very  large,  and  ouite  time-consuming  to  compute.   A 
simpler  function  is: 

G(i1,i2,  ...  in)  =  2  x  +  2  c   +  ...  +  2  n  (B) 

which  is  invertible  if  none  of  the  i's  are  duplicated,  as  is: 

i-,    I,,  i 

G(ilfi2,  ...ln)=2x^2c^...^2n  (C) 

where  4   is  the  Boolean  'non-eouivalence'  operation. 
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If  there  are  duplicate  i's,  but  they  are  not  significant, 
the  function 

G(i1,i2,  ...  ln)  =  2XlV  2  2V  ...  V  2  n        (D) 
can  be  used,  where  V  is  the  Boolean  'inclusive  or'  operation. 

With  regard  to  the  inclusion  test,  we  can  only  infer  that 
S-,  is  included  in  S?   from  the  relationship  between  H(S,)  and 
H(S?)  if  once  again  h  has  an  inverse.   However,  the  relation 
between  H(S,)  and  H(S2)  is  no  longer  equality,  but  depends  on 
the  particular  operator   used.   For  (A)  above  a  necessary 
and  sufficient  condition  is  that  E(Sj)    divide  H(Sg)  exactly. 
This  is  once  again  computationally  inferior  to  the  other  func- 
tions.  The  function  (d)  can  be  used  if  there  are  duplicate 
i's,  with  the  necessary  and  sufficient  condition  for  inclusion 
being  that  H(SO  is  implied  by  H(S2),  or,  more  simply,  that 
H(S2)  has  bits  wherever  H(S1)  has  bits.   If  there  are  duplicated 
i*s,  the  functions  (B)  and  (C)  cannot  be  used  for  the  inclusion 
test,  but  if  the  i's  are  unique,  either  can  be  used  in  the  same 
way  as  (D).   In  what  follows  we  will  restrict  consideration  to 
function  (D)  above  unless  explicitly  stated  otherwise. 
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The  Use  of  Hashing 

The  simple  scheme  we  have  considered  includes  the  well- 
known  method  for  implementing  set  operations  which  represents 
each  possible  element  as  a  unique  integer,  and  each  set  as  a 
string  of  bits  having  ones  corresponding  to  elements  which  are 
present  in  the  set,  and  zeros  elsewhere.   The  disadvantage  of 
this  method  is  that  it  requires  one  bit  for  each  element, 
regardless  of  the  actual  size  of  the  sets  being  considered.   In 
fact,  for  large  universes  of  elements  and  small  sets  this  can 
be  less  efficient  than  the  search  procedure  described  before. 

What  we  will  suggest  here  is  that  the  method  described 
above  is  still  useful  even  when  the  function  h  does  not  have 
an  inverse.   In  this  case  the  function  H  also  has  no  inverse, 
and  there  may  be  many  sets  which  are  mapped  by  H  into  the  same 
value.   Nevertheless,  it  is  still  possible  to  extract  useful 
information  from  such  a  test  if  the  appropriate  functions  are 
used.   For  instance,  whatever  the  function  h,  if  H(S, )  and 
H(S2)  are  unequal,  the  sets  S,  and  S?  cannot  be  identical. 
Similarly,  if  H(S1)  is  not  implied  by  H(S2),   then  S,  is  not 
included  In  Sp. 

In  what  follows  we  will  concern  ourselves  with  a  particular 

set  function: 

h(e.) 
H(S)  =2    1 

for  some  appropriately  chosen  hash  function  h  which  produces 


values  in  the  range  0  to  M-l.   Thus  the  function  H  produces 
an  M-bit  value  for  each  set  which  we  will  call  the  'signature' 
of  the  set. 

If  we  assume  that  the  hash  function  h  is  random,  then 
the  distribution  of  the  number  m  of  zeros  in  the  signature  of 
a  randomly  chosen  set  of  n  elements  is  the  same  as  the  distri- 
bution of  the  number  of  empty  cells  out  of  M  after  a  random 
allocation  of  n  balls  with  replacement  from  a  total  population 
of  N.   For  large  N,  the  case  we  are  interested  in,  this  is 
approximately  eaual  to  the  distribution  without  replacement. 
In  the  limit  as  n  and  M  tend  to  infinity  with  o=nMf    the 
probability  of  finding  a  cell  containing  k  balls  is  (Feller 

p. 58,  ex.6) : 

-a  k  a  . 
Pk  =  e  uq  A! 

Thus  the  probability  of  finding  a  cell  empty  is: 

-n/M 
P0  "  e 

and  the  expectation  value  of  the  number  of  zeros  is  Me~    . 

For  maximum  information  content,  we  would  expect  the 
expected  density  of  zeros  to  be  \,    i.e.: 

^M  =  Me"n/M 
so  n/M  =  In  2,  and  we  should  choose  M  to  be  n/ln2,  somewhat 
larger  than  n. 
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ORDERED  SETS 

The  same  sort  of  techniques  can  also  be  applied  to 
ordered  sets,  that  is,  sets  of  elements  whose  order  is 
significant.   Such  sets  occur  frequently  in  all  areas  of 
information  processing,  the  simplest  being  a  string  of  charac- 
ters. 

One  way  of  constructing  the  hash  code  corresponding  to 
an  ordered  set  which  preserves  information  about  the  ordering 
is  to  use  a  hash  function  h  on  an  element  which  is  a  function 
of  the  element  and  its  position.   Thus  we  can  write: 

h(e±)  =  h(e1,i) 

This  permits  answers  to  ouestions  of  the  form:   does  ordered 

set  S  contain  the  elements  e,,  e0  ...  e„  in  positions  i,, 

id  ra    c  1 

i?,  . . .  i  .   However,  it  does  not  permit  an  efficient  implement- 
ation of  a  test  for  a  sequence  of  characters  occurring  consecu- 
tively in  a  specified  order  anywhere  within  the  given  ordered 
set.   This  of  course  includes  the  problem  of  determining  if  a 
word  occurs  in  a  sentence. 

A  more  useful  technique  is  to  represent  the  ordered  set 
by  the  set  of  subsequences.   This  latter  set  is  of  course  not 
ordered,  so  the  techniques  given  above  for  such  sets  can  be 
used.   It  is  not  necessary  to  use  all  subsequences,  and  in  fact 
the  method  which  seems  preferable  is  to  use  all  seauences  of 
a  particular  length.   Thus  we  might  represent  an  ordered  set 
S,  of  n  elements  by  the  n-1  two-element  sequences,  or  by  the 
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n-2  three- element  sequences.   A  test  for  a  particular  subse- 
quence Sp  then  might  take  the  form  of  three  tests: 

1.  Test  S,  for  the  elements  of  Sp. 

2.  Test  S-,  for  the  two-element  sequences  of  Sp. 

3.  Test  S-,  for  the  three-element  sequences  of  S?. 
If  S,  does  not  include  all  one-,  two-,  and  three-element 
subsequences  of  S?,  then  the  sequence  S2  does  not  occur  in  S..  . 
Accordingly,  failure  of  any  of  the  three  tests  would  constitute 
failure  overall. 

We  will  refer  to  a  subsequence  of  k  successive  elements 
of  a  string  as  a  k-sequence. 

Let  us  consider  in  more  detail  the  representation  of  a 
string  by  its  set  of  2-sequences.   If  there  are  N  different 
symbols  in  the  alphabet,  we  note  that  there  are  In  possible 
2-sequences.   We  will  use  the  term  k-signature  of  a  string  to 
refer  to  a  sequence  of  n     bits  in  which  each  bit  is  one  if 
the  corresponding  k-sequence  is  present,  and  zero  otherwise. 
If  there  are  n  symbols  in  the  string,  there  will  be  n-1 
2-sequences,  so  when  n  is  small  compared  with  n,  the  density 
of  ones  in  the  2- signature  is  small.   On  the  other  hand,  when 
n  is  large  compared  with  w  ,  we  will  have  some  2-sequences 
occurring  more  than  once,  and  the  average  density  of  ones  will 
be  closer  to  1.   As  n  gets  very  large  compared  with  n,    we  will 
find  that  nearly  all  strings  have  all  2-sequences  in  them,  so 
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that  most  2-signatures  will  be  all  ones.   We  expect  to  find 
maximum  information  in  a  2-signature  when  there  are  eaual 

numbers  of  ones  and  zeros,  and  we  will  expect  this  for  a  value 

2 
of  n  approximately  equal  to  N  . 

Unfortunately,  we  cannot  say  that  strings  constructed 

randomly  from  an  alphabet  of  N  symbols  have  2-signatures 

which  come  from  a  population  which  is  constructed  randomly  from 

2 
an  alphabet  of  N  2-sequences.   In  fact,  there  are  some  2-signa- 
tures which  do  not  correspond  to  proper  strings  at  all.   Thus 
we  cannot  assume  a  Poisson  distribution  of  ones  in  2-signatures 
for  random  strings.   However,  a  few  calculations  indicate  that 
the  error  in  making  this  assumption  is  probably  quite  small, 

particularly  in  the  region  we  are  interested  in  where  n  is  close 

2 
to  N  .   In  the  table  below  we  give  the  numbers  of  ones  in 

2-signatures  of  100  random  strings  chosen  from  a  4-character 

alphabet  for  various  lengths  of  strings.   It  is  seen  that  for 

n  =  16,  the  average  number  of  ones  in  a  2-signature  is  about 

10.0,  and  the  Poisson  distribution  v;ould  give  l6(l-e~  )  ~  10.0 

also. 
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length 

of 
strings 

b 

8 
16 
32 

64 

128 


number  of  bits  in  2-signature 


1 

2   3 

4  5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

1 

l4  85 

0   0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  0 

8  26 

4o 

26 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  0 

0  0 

2 

2 

10 

21 

28 

21 

13 

3 

0 

0 

0 

0 

0   0 

0   0 

0 

0 

0 

0 

0 

3 

8 

19 

33 

33 

b 

0 

0   0 

0   0 

0 

0 

0 

0 

0 

0 

0 

1 

2 

30 

67 

000000000   0   0   0   0   0   199 

Distribution  of  the  number  of  bits  in  2-signatures, 


If  the  strings  are  much  longer  than  w  ,  a  2-signature  will 

give  little  information,  and  we  should  choose  a  3-signature. 

3 
This  will  be  good  for  strings  with  lengths  up  to  about  N  , 

while  much  longer  strings  will  require  a  4-sequence,  and  so  on. 

What  we  are  suggesting,  therefore,  is  a  representation  of  a 

string  of  n  symbols  by  a  bit-string  whose  length  is  between 

n  and  nN  bits. 

It  is  clear  that  when  an  ordered  set  (string)  is  represented 
by  its  set  of  k-sequences,  the  subsequence  test  on  the  strings 
becomes  the  subset  test  on  the  sets  of  k-sequences.   This  means 
that  we  can  use  the  hashing  techniques  mentioned  in  previous 
sections  for  implementing  the  subseauence  test.   The  resulting 
procedure  represents  a  string  by  a  "hashed  k-signature"  obtained 
as  follows: 

For  each  k-sequence  of  the  string,  obtain  an  integer 
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1  £  i  <  M  by  applying  a  hashing  function,  and  set 

bit  i  of  the  M-bit  hashed  k-signature  to  1.   Set  all 

other  bits  to  zero. 
This  permits  the  representation  of  the  string  by  a  bit-string 
of  arbitrary  length  M.   The  value  of  k  for  given  M  should  be 
chosen  so  that  the  number  of  bits  in  the  hashed  k- signature 
is  close  to  M/2.   This  implies  that  the  number  of  k-sequences 
be  close  to  Mln2,  so  we  should  have: 

n-k+1  ~  M  In  2 
In  practice  k  has  to  be  smaller  than  the  length  of  the  shortest 
string  which  will  be  encountered,  so  we  can  usually  neglect  it, 
and  choose  M  «  n/ln2,  as  before. 

This  rather  curious  result  suggests  the  same  number  of 
bits  be  used  to  represent  an  unordered  and  an  ordered  set. 
Clearly  more  information  is  required  to  specify  the  ordered 
set,  so  our  representation  of  the  ordered  set  is  less  accurate. 
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EXAMPLES 

To  illustrate  the  utility  of  the  methods  suggested 
above,  a  Fortran  program  was  written  to  do  the  inclusion 
test.   The  members  of  the  sets  were  the  6-bit  characters 
A-Z  represented  by  integers  1-26  and  the  sets  chosen  were 
the  following  words: 

A,  JOB,  CONSISTS,  OF,  ONE,  FILE,  PUNCHED, 

CARDS,  OR,  CARD,  IMAGES,  THE,  FIRST,  LOGICAL, 

RECORD,  CONTROL 
These  were  the  distinct  words  in  the  first  two  random  sentences 
of  a  manual  which  was  at  hand.   All  pairs  of  words  were  compared 
for  inclusion  using  M0D(l,15)  as  the  hashing  function  for  a 
character,  thus  giving  a  15-bit  hash  code  for  each  word.   Of 
the  256  comparisons  made  only  10  were  in  error.   The  23  inclusions 
were  of  course  all  identified,  namely,  the  16  identical  pairs, 
and  also  A  in  CARDS,  CARD,  IMAGES,  and  LOGICAL,  OR  in  RECORD 
and  CONTROL,  and  CARD  in  CARDS.   The  errors  were  that  A  was 
said  to  be  in  PUNCHED,  ONE  in  CONSISTS  and  LOGICAL,  CARD  in  PUNCHED, 
THE  in  PUNCHED,  and  RECORD  in  LOGICAL.   If  the  size  of  the  set 
was  used  to  eliminate  those  comparisons  which  could  not  be  true, 
the  total  number  of  comparisons  would  have  been  146,  and  there 
would  have  been  9  errors. 
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A  second  test  was  made  with  the  same  sets  but  assumed 
to  be  ordered,  so  that  inclusion  would  require  that  one  string 
be  a  subsequence  of  the  other.   For  this  test  the  3-letter 
subsequences  of  each  string  were  hashed  using  M0D(l,15), 
once  again  giving  a  15-bit  hash  for  each  word.   The  one-  and 
two-letter  words  were  not  used  for  this  test,  and  out  of  the 
169  pairs  the  13  identical  pairs  and  CARD  in  CARDS  were,  of 
course,  identified.   10  errors  were  made,  JOB  being  given  as 
a  subsequence  of  THE,  ONE  of  IMAGES,  LOGICAL,  RECORD,  and 
CONTROL,  PILE  of  CONSISTS,  PUNCHED,  and  IMAGES,  CARD  of  PUNCHED, 
and  THE  of  JOB.   However,  note  that  if  this  comparison  had  been 
followed  by  the  first  comparison  in  the  case  of  possible  inclu- 
sion, only  2  errors  would  have  been  made. 

When  the  above  two  tests  were  repeated  using  a  31-bit 
hash  code  instead  of  15-bit,  the  first  test  gave  no  errors  and 
the  second  gave  3  errors  instead  of  10.   The  first  result  is, 
of  course,  a  direct  consequence  of  the  fact  that  the  31-bit  hash 
code  is  invertible  for  the  characters  used. 

A  more  complete  application  was  foind  to  the  implemen- 
tation of  an  on-line  editor,  which  was  to  permit  lines  to 
be  located  by  searching  for  a  seouence  of  characters  speci- 
fied by  the  user.   By  storing  an  additional  hash  word  with 
each  line,  this  search  can  be  speeded  up  considerably.   A 
test  of  this  procedure  was  undertaken  on  the  Fortran  text 
given  below. 
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PROGRAM    SET  HASH    (TAPE   1,     TAPE  2,     OUTPUT1! 

DIMENSION  KARS(72) 

DIMENSION  I  WORD  (10) 
C    BUILD  PATTERN  HASH 
1000  READ  (2,101*)  I  WORD 

104  FORMAT    (10R1^ 

IF    (ENDFILE   2)    2000,6 
2000    CALL   EXIT 
6  DO   1   1=3,10 

IF    (I  WORD    (1).    EQ.    I  WORD    (i) )    GO  TO   2 

1  CONTINUE 
1=11 

2  NCH=I-2 

IPATT=KHASH(I  WORD    (2),NCH) 
PRINT   105  >    IPATT 

105  FORMAT    (*1*,020) 
C  BUILD   STRING  HASH 

REWIND   1 

99  READ    (1,100)    KARS 

100  FORMAT    (72R1) 

IF    (ENDFILE   1^    1000,5 
5  ISTRING=KHASH    (KARS, 72) 

C  COMPARE 

IF    (     (IPATT.A.ISTRING) .EQ. IPATT)    GO   TO   *) 

PRINT   102,   KARS,    ISTRING 
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102   FORMAT  (#  *,72R1, 8X, 020) 

GO  TO  99 
4     PRINT  101,  KARS, I STRING, IWORD 
101   FORMAT  (*  *  ,72R1,8X,020,  lOx,  10R1) 

GO  TO  99 

END 

FUNCTION  KHASH(KARS, L) 

DIMENSION  KARS(72^ 
C  DEFINE  HASH  FUNCTION 

IHASH(lUMOD(l,  45) 
C  BUILD  HASH 

KHASH=0 

DO   ?    1=2,  L 

IPR=I    GHIFT(KARS(l-l)  .6)  .O.KARS(l) 

KHASH=KHASH.O.  (2**I   HASH(lPR)) 

3     CONTINUE 
RETURN 
END 

A  search  for  the  identifier  KARS  using  2-seouences  and 
a  31-bit  hash  found  exactly  the  correct  lines,  while 
searches  for  PRINT  gave  the  three  occurrences  and  two 
"errors",  for  FORMAT  gave  the  five  occurrences  and  three 
errors.   On  the  other  hand,  a  search  for  a  two-character 
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senuence  IF  gave  the  four  occurrences  and  twenty-two 
errors,  which  might  have  been  expected  from  a  1-bit 
hash  code.   The  search  for  IF  preceded  and  followed  by 
a  blank  cut  the  errors  down  to  two. 

When  the  hash  was  increased  to  ^5  bits,  the  results 
improved  with  KARS,  PRINT  and  blank-surrounded  IF  giving 
no  errors,  FORMAT  two  instead  of  three  errors,  anri  IF  ten 
instead  of  twenty- two  errors. 

The  Fortran  text  given  above  is  of  course  the  program 
used  to  dc  the  test.   It  is  not  intended  to  be  taken  as  a 
recommended  implementation. 
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An  Application  to  Information  Retrieval 

There  is  a  simple  application  to  the  problem  of  information 
retrieval,  which  we  will  pose  in  the  following  form.  Suppose 
we  have  a  large  number  of  documents  characterized  by  an  arbi- 
trary number  of  attribute-value  pairs,  which  can  be  considered 
to  be  strings  of  coded  characters  or  numeric  auantities.  The 
retrieval  problem  takes  the  form  of  reauesting  that  all  docu- 
ments which  have  certain  values  for  certain  attributes. 

We  will  store  the  information  in  the  following  way.   Consider 
each  document  as  a  set  of  attribute-value  pairs,  and  hash  each 
pair  into  a  number  between  0  and  M-l  for  some  M  which  might 
typically  be  the  number  of  bits  in  a  machine  word.   Calculate 
the  signature  for  each  document.   This  will  be  an  M-bit  auantity. 
Store  the  documents  on  secondary  storage  so  that  documents  with 
the  same  signature  are  stored  together.   Construct  in  memory  a 
table  giving  the  address  of  the  first  document  with  each  signa- 
ture.  Not  all  possible  signatures  may  occur,  of  course,  so  this 
table  may  be  sparsely  occupied,  reouiring  a  tree  structure  or 
simply  a  sorted  list  for  fast  lookup.   The  retrieval  algorithm 
then  takes  the  following  form.   Calculate  the  signature  R  of 
a  document  which  contains  just  the  required  attribute-value 
pairs.   All  documents  which  contain  these  pairs  will  have  a 
signature  which  will  contain  bits  where  R  goes.   All  documents 
with  such  signatures  are  retrieved. 


21 


To  get  some  feeling  of  the  reduction  in  the  amount  of 
searching  which  is  possible  by  this  method,  suppose  M=32  and 
the  number  of  attribute-value  pairs  required  is  4.   In  the 
majority  of  cases  this  will  generate  a  4-bit  signature,  so 
only  documents  with  these  4  bits  set  in  their  signature  should 

be  examined.   This  reduces  the  number  of  possible  signatures 

"52     28  4  r 

from  2^   to  2   ,  a  gain  of  a  factor  of  2  =16  in  number  of 

signatures.   With  a  random  distribution  of  signatures  this 

would  also  reduce  the  number  of  documents  to  be  searched  by 

16  also.   In  fact,  this  distribution  will  not  be  random,  so 

performance  will  usually  be  much  better  than  this.   For  instance, 

if  each  document  is  restricted  to  m/attribute-value  pairs,  the 

number  of  signatures  will  be  restricted  to  (M) .   In  our  example, 

if  p=5  there  will  be  (32.31.30.29.28)/(5.4.3.2.1)  signatures, 

which  Is  about  2000.   Of  these,  there  will  be  only  29  acceptable 

signatures,  giving  a  reduction  by  a  factor  of  about  70. 
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